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Vh ' Abstract. We define and study supercharacters of the classical finite unipotent groups 

of types B n (q), C n (q) and D n (q). We show that the results proved in [7] remain valid 
over any finite field of odd characteristic. In particular, we show how supercharacters 
r — . for groups of those types can be obtained by restricting the supercharacter theory of 

£S) ' the finite unitriangular group, and prove that supercharacters are orthogonal and pro- 

vide a partition of the set of all irreducible characters. In addition, we prove that the 
unitary vector space spanned by all the supercharacters is closed under multiplication, 
and establish a formula for the supercharacter values. As a consequence, we obtain the 
decomposition of the regular character as an orthogonal linear combination of super- 
characters. Finally, we give a combinatorial description of all the irreducible characters 
of maximum degree in terms of the root system, by showing how they can be obtained 
as constituents of particular supercharacters. 






)f2 ■ 1- Introduction 



The concept of a "supercharacter theory" for an arbitrary finite group was developed by 
P. Diaconis and I. M. Isaacs in the paper [15]. Roughly, a supercharacter theory replaces 
irreducible characters by "supercharacters", and conjugacy classes by "superclasses", in 
such a way that a "supercharacter table" can be constructed as an "almost unitary" 
matrix with similar properties as the usual character table (namely, orthogonality of 
rows and columns). More precisely, given any finite group G, a supercharacter theory for 
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G consists of a partition /C of G and a set X of (complex) characters of G satisfying the 
following three axioms: 



(i) \K\ = \X\; 

(ii) every irreducible character of G is a constituent of a unique £ G X 
(hi) the characters in X are constant on the member of JC. 



i 



The elements of JC will be referred to as superclasses, and the elements of X as superchar- 
acters of G. (We observe that, by [TSJ Lemma 2.1], axiom (ii) is equivalent to requiring 
that {1} G JC.) 

Every finite group G has two "trivial" supercharacter theories: the full character theory 
(where X consists of all irreducible characters of G, and JC of all the conjugacy classes of 
G), and the one where X = {1g,Pg — If?} and K consists of the sets {1} and G — {1}; 
as usual, we denote by 1q the trivial character and by pa the regular character of G. 
Although for some groups these are the only possibilities, there are many groups for 
which nontrivial supercharacter theories exist, and in many cases it may be possible to 
obtain useful information using some particular supercharacter theory. An illustrating 
example can be found in the paper [11] where E. Arias-Castro, P. Diaconis and R. Stanley 
showed that a special supercharacter theory can be applied to study a random walk on 
upper triangular matrices over finite fields using techniques that traditionally required 
the knowledge of the full character theory. 

Supercharacters theories were initially developed for the upper unitrangular group 
U n (q) consisting of all unipotent upper-triangular n x n matrices over the finite field ¥ q 
with q elements (where q is a power of some prime number p) . It is known that an explicit 
description of the irreducible characters and conjugacy classes of U n (q) is an "intractable" 
problem; in fact, in the paper [16] . Gudivok et al. show that a "nice" description of the 
conjugacy classes of U n (q) leads to a "nice" description of wild quivers. However, in 
his PhD thesis [I], the first author begun the study of the "basic characters" of U n (q) 
(under the assumption that p > n), and was able to show that "clumping" together some 
of the conjugacy classes and some of the irreducible characters one attains a workable 
"approximation" to the representation theory of U n (q). His results were published in 
a series of papers in the Journal of Algebra, and showed in particular that the "basic 
characters" determine uniquely the superclasses of a supercharacter theory for U n (q) (the 
"theory of basic characters" was renamed after Roger W. Carter as "a superclass and 
supercharacter theory"). We mention that the original theory relies on a construction 
due to D. Kazdhan (see [19]) and is based on Kirillov's method of coadjoint orbits (see 
[20] for a description of Kirillov's method for the unitriangular group; see also [22] where 
J. Sangroniz gives a general version of Kazdhan's construction for algebra groups defined 
over finite fields of sufficiently large characteristic). Later, in his PhD thesis [25], N. Yan 
showed how the "basic characters" can be obtained using more elementary methods which 
avoid Kazhan's construction and the algebraic geometry involved in it. Yan's approach 
is valid for an arbitrary prime, and was generalized later by P. Diaconis and M. Isaacs in 



the paper [15] in order to extend the theory for an arbitrary finite algebra group defined 
over ¥ q (see also [9] where a generalization was obtained for algebra groups defined over 
finite radical rings and over certain rings of p-adic integers). 

The main goal of this paper is to extend to an arbitrary odd prime the results obtained 
in the paper [7] where the authors started to develop a supercharacter theory for a Sylow 
p-subgroup U of one the (non- twisted) Chevalley groups C n (q), B n (q), and D n (q). (We 
mention that the present paper is a companion of the forthcoming paper [8] where a 
supercharacter theory for U is established by defining the superclasses of U.) As in [7], 
the notion of a supercharacter of U is very similar the notion of a "basic character" of 
the unitriangular group U n (q), and follows the original idea of parametrizing superchar- 
acters by certain "minimal" subsets of (positive) roots. In fact, it is known that the 
supercharacters of U n (q) can be obtained as certain "reduced" products of "elementary 
characters" which are irreducible characters corresponding to the "matrix entries" (i,j), 
for 1 < i < j < n, labelled by nonzero elements of ¥ q ; in Yan's thesis, the "elemen- 
tary characters" were called "primary characters", and the supercharacters were called 
"transition characters" . (We refer that the factorization of a supercharacter as a product 
of "elementary characters" holds, not only for the unitriangular group, but also for any 
finite algebra group, as it is explained in the paper [10J where O. Pinho and the first 
author obtain a relation between factorizations of supercharacters and decomposability 
of certain cyclic modules.) 

Following Yan's method, one can show that the supercharacters of U n (q) are parametri- 
zed by certain combinatorial data consisting of a "basic set" D of matrix entries such that 
no two elements of D agree in, either the first, or the second, coordinate, and of a map </> 
from D to the nonzero elements of ¥ q . (There is an alternative way of parametrizing the 
supercharacters of U n (q) by labelled set partitions of {1, 2, . . . , n}, and we mention that 
a rich combinatorial structure is arising and appears to have a remarkable analogy with 
the well-known connection between partitions of n and the representation theory of the 
symmetric group S n ; see the papers [2TJ [21] by N. Thiem and his collaborators.) 

In the present paper, as in [7J, we define the supercharacters also as certain "reduced" 
products of "elementary characters" (which in general are not necessarily irreducible 
characters) of the given Sylow p-subgroup U. These "reduced" products are parametrized 
by pairs consisting of a conveniently chosen "basic subset of roots" and of a map to the 
nonzero elements of ¥ q . (We note that the roots in the unitriangular case are in one- 
to-one correspondence with the matrix entries.) In fact, the group U can be naturally 
identified with a subgroup of a unitriangular group, and we will show that the elementary 
characters (and supercharacters) of U can be obtained as constituents of the restriction 
of a supercharacter of that unitriangular group. 

The paper is organized as follows. In Section [2], we introduce the necessary notation 
and define the elementary characters and the supercharacters of the group U. Then, in 
Section [31 we obtain the elementary characters of U by restricting elementary characters 



of the unitriangular group which contains U, and use this information to show that the 
complex vector space spanned by the supercharacters is, in fact, the associative algebra 
(finitely generated) by the elementary characters. As a consequence, we deduce that 
every irreducible character of U is a constituent of a supercharacter. Then, in Section 
we prove the orthogonality of supercharacters (as class functions of U), by using a 
partition of the dual space of the Lie algebra of U in terms of its "basic subvarieties" as 
obtained in Theorem 4.5 of the authors' paper [7j. In Section [6], we deduce a formula 
for the supercharacters analogous to the one proved in [151 Theorem 5.6], and obtain a 
decomposition of the regular character of U as a linear combination (with nonnegative 
integer coefficients) of supercharacters. Finally, in Section [7J we apply our results on 
supercharacters to identify the irreducible characters of maximum degree of U. We 
observe that, for several times, we refer to results proved for the unitriangular group 
under the assumption that the prime p is sufficiently large; however, those results are 
known to be true for arbitrary primes, as it follows from Yan's work (see also [5J, or [15J). 

2. Supercharacters 

Let p > 3 be a prime number, q = p e (e > 1) a power of p, and ¥ q the finite field 
with q elements. For a fixed positive integer n, let G denote one of the following classical 
finite groups: the symplectic group Sp2 n (q), the even orthogonal group 02n(<?), or the odd 
orthogonal group 02 n +i {q) (in alternative notation, these are the (non- twisted) Chevalley 
groups C n (q), B n (q), and D n (q), respectively). Throughout the paper, we set U = 
G fl U m (q) where 

if G = Sp 2n (q), or G = 2n (q), 
if G = 2n +i(q)„ 

and U m (q) denotes the upper unitriangular group consisting of all unipotent upper- 
triangular m x m matrices over ¥ q . Then, U is a Sylow p-subgroup of G, and it is 
described as follows. Let J = J n be the n x n matrix with l's along the anti-diagonal 
and O's elsewhere. Then, U consists of all (block) matrices of the form 

(X xu xz 
I r -u T J 
Jx' T J / 

where x £ U n (q), u is an n x r matrix over ¥ q , and 

(i) r = 0, and Jz T — z J = 0, if U < Sp2 n (q)', 
(ii) r = 0, and Jz T + zJ = 0, if U < 2n {q)\ 
(iii) r = 1, and Jz T + zJ = —uu T , if U < 02„+i(g). 

As mentioned in the Introduction, the supercharacters of U will be parametrized by 
certain subsets of (positive) roots. Thus, we introduce some notation and recall some 
elementary facts concerning roots; for the details, we refer to the books [121 US] by R. 




Carter (see also [TH Chapter 8]). Let T be the maximal torus of G consisting of all 
diagonal matrices, and £ the root system defined by T. The elements of £ are described 
as follows. For each 1 < i < n, let £^: T — ► F x be the map defined by Si{t) = t t for 
all t G T; here, we denote by ti G F x the (i,i)th entry of the matrix t G T. Then, 
£ = $ U (-$) where 

$ = {e i ±s j : 1 < i < j < n} U $' 

and 

f{2e i: l<i<n}, ifG = Sp 2 „(g), 

$'= <0, ifG = 2 „(g), 

{{ei:l<i<n}, if G = 2n+1 (q). 

The roots in $ are said to be positive, and the roots in — $ are said to be negative. 
Throughout the paper, the word "root" will always stand for "positive root" . 

With $ we associate the subset of "matrix entries" £ C {(i,j): — n < i, j < n} as 
follows. For any a G $, we set 

'{(iJ)> (-i; -«)}> if a = Sj - £j for 1 < z < j < n, 

{(», -J)> (i) -«')}» if a = £i + £j for 1 < i < j < n, 

{(i, —i)}, if G = Sp 2 n(q) and a = 2ei for 1 < i < n, 

k {(i, 0), (0, -i)}, if G = 2 „+i(g) and a = ei for 1 < i < n, 



S(a) 



and we define 



£={j£( 



(n) 
More generally, for each subset \I/ C $, we set 



£(*) = u^( 



hence, £ = £($). 

On the other hand, we consider the mirror order -< on the set {0, ±1, . . . , ±(n + 1)} 
which is defined as 

1^2^ < n + 1 ■< -< -{n + 1) -< < -2 -< -1, 

and we shall index the rows (from left to right) and columns (from top to bottom) of any 
mxm matrix according to this ordering. Hence, the entries of any matrix x G U m (q) are 
indexed by all the pairs (i,j) G £ : for each (i,j) G £, we shall write x^j to denote the 
(i,j)th entry of x (which occurs in the ith row and in the jth column). For our purposes, 
it is convenient to consider the set 

£+ = {(jj) e£:l<i<n,i-<j^ -i}, 

and extend this notation to any subset \I/ C $ by setting 

£+(^) = £(q)n£ + . 



We observe that there exists a one-to-one correspondence between $ and £ + . 
For any a G $, we define the subgroup U a of U as follows: 

(i) if a = Ei — Ej for 1 < i < j < n, then 

U a = {x G U: x i<k = 0, i < k < j}; 

(ii) if a = Ei — Ej for 1 < i < j < n, then 

U a — {x G U: x it k = Xj t i = 0, i < k < n, j -< / ^ 0}; 

(iii) if, either a = 2&j for 1 < i < n (in the case where U < Sp2 n {<l)), or a = Ei for 
1 < i < n (in the case where C/ < 2n+ i(g)), then 

U a = {x G U: Xi^ = 0, i < k < n}. 

Let $ : F q — > C x be a non-trivial linear character of the additive group F 9 + of F^ (this 
character will be kept fixed throughout the paper; moreover, all characters will be taken 
over the complex field). For any r G F x , the mapping x i— > d(rxij) defines a linear 
character A Qjr : U a — > C x of U a , and we define the elementary character £ a)r to be the 
induced character 

(see P] for the corresponding definition in the case of the unitriangular group; see also 
[T5| Corollary 5.11] and the discussion thereon). 

We next define the notion of a "basic subset of roots" . To start with, we recall that a 
subset T> C £ is said to be basic if it contains at most one entry from each row and at 
most one root from each column; in other words, T> C £ is basic if 

\{j:i<j±-l, (i,j)eV}\<l and \{i: 1 z< % -< j, (i, j) G V}\ < 1 

for all — n < i, j < n. Then, we say that D C $ is a 6aszc subset ii T> = £(D) is a basic 
subset of £ . (We will always use script letters to denote basic subsets of £, in contrast 
to basic subsets of $ which will be mostly denoted by italic letters.) 

Given any non-empty basic subset D C $ and any map <fi: D —>■ F x , we define the 
supercharacter £ D ^ to be the product 

a<=D 

For convenience, if D is the empty subset of $, we consider the empty map 0: D — > F x , 
and define ^d,^> to be the unit character lfy of U. Let 

Ud = f j U a and A^^ = ] | (A^^q,))^. 

Then, A^ ^ is clearly a linear character of Ud and, by [7, Proposition 2.2], the superchar- 
acter ^ D ^ can be obtained as the induced character 

(2b) 6^ = (Xd^) U . 



a 


u 


w 
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-u T J 








-Ja T J 



We now state the main result of this paper which extends pj Theorem 1.1] for arbitrary 
odd primes. (Given any finite group G, we denote by Irr(G) the set of all irreducible 
characters of G, and by (-, •) (or by (-, -)q if necessary) the Frobenius' scalar product on 
the complex vector space of all class functions defined on G.) 

Theorem 2.1. Let x be an arbitrary irreducible character ofU. Then, x is a constituent 
of a unique supercharacter ofU; in other words, there exists a unique basic subset D C $ 
and a unique map (ft: D — > F g x such that (x,£d,4>) ¥" 0- 

Our proof depends strongly on the supercharacter theory of the unitriangular group, 
and on certain "basic subvarieties" defined by polynomial equations on the dual space of 
the Lie algebra u of U. We recall its definition. Let g denote one of the following classical 
Lie algebras defined over W q : the symplectic Lie algebra sp 2n (q), the even orthogonal 
Lie algebra o 2n (q), or the odd orthogonal Lie algebra o 2n+ i(q). Then, u = q fl u m (q) 
where u m (g) denotes the upper niltriangular Lie algebra consisting of all nilpotent upper- 
triangular m x m matrices over ¥ q . Thus, u consists of all (block) matrices of the form 

(2c) 

where a G u n (q), u is an n x r matrix over ¥ q , and 
(i) r = 0, and Jw T — wJ = 0, if u < sp 2n (q); 
(ii) r = 0, and Jw T + wJ = 0, if u < 02 n (q)', 
(iii) r = 1, and Jw T + wJ = —uu T , if u < o 2n+ i(g). 

For any a G $, we will denote by e a the matrix in u defined as follows (as usual, 
1 < i < j < n): 

e i~j + e j -ii if a = £ i + £ j an d U ^ S p2n( ( l)i 

e i-j - e j-ii if a = £ i + £ j and u < o 2n {q) or u = o 2n +i(q), 
ei-i, if u < sp 2n (q) and a = 2£j, 

^e ifi - e -i, if u < o 2n +i(q) and a = £,. 

It is clear that {e a : a G $} is an F^-basis of u. On the other hand, we denote by u* 
the dual vector space of u, and let {e* : a G $} be the F^-basis of u* dual to the basis 
{e a : a G $} of u; hence, e* (ep) = 5 a> p for all a, (3 G <&. 

For any a G $, we define the Lie subalgebra u a of u as follows: 
(i) if a = Si — Ej for 1 < i < j < n, then 

u a = {a G u: Oj jfe = 0, i < k < j}; 

(ii) if a = Ei + £j for 1 < i < j < n, then 

u a = {a G u: aj^ = a^ = 0, i < k < n, j -< / ^ 0}; 



(iii) if, either a = lei for 1 < % < n (in the case where u < sp 2n (g)), or a = s^ for 
1 < i < n (in the case where u < o 2n+ i(g)), then 

u Q = {a G u: a ijk = 0, i < k < n}. 

We note that 

u a = ^2 F <? e £ 

/3£*(a) 

where $(a) = {(3 G $: e^ G u a }; hence, {e^: /5 G $(a)} is a basis of u a . 

Remark 2.2. In the case where p > 2n, we have a p = for all a G u, and so we may define 
the usual exponential map exp : u — > U by exp (a) = 1 + a + h a 2 + • • ■ + ^a n for all a G u. 
It is well-known that exp is bijective and that the Campbell-Hausdorff formula holds: for 
all a, b G u, we have exp(a) exp(6) = exp(a + b + i?(a, 6)) where $(a, 6) G [u, u] (see [18j 
pg. 175]). It follows that, if f) is any Lie subalgebra of u, then the exponential image 
H = exp(f)) is a subgroup of U. In particular, for any a e $, we have U a = exp(u Q ). 

3. Elementary characters 

We start this section by relating the elementary characters of U with the elementary 
characters of the corresponding unitriangular group U m (q). Firstly, we fix some notation. 
To avoid confusion, we shall denote by Cij> the elementary character of U m (q) associated 
with the entry (i,j) G £ and the element r G F g x . By the definition, £jj >r . is the induced 
character 

C- ■ = (n, ■ ) u ^) 
where /ijj, r : U it j — »■ C x is the linear character of the subgroup 

Uij = {x G U m (q) : x i)k = 0, i -< k -< j} 
defined by /J*i,j, r (x) = ^{rxij) for all x G Uij. We observe that, if 1 ^ i -< j ^ and 



a = Si — Ej, then 



U itj nU = U a and U itj U a = U m (q); 



for simplicity of writing, we set e = 0. For the remaining cases, the following lemma 
will be useful. 

Lemma 3.1. Let (i, —j) G £ for 1 < % < j < n and let r G F g x . Let 

U[-j = {x G U m (q) ■■ %i,b = x-a-j = 0, i -< b ^ 0, j -< a ^ 0}, 

and let Vi-j >r '- U[ ^ — » C x 6e £/ie linear character of\J[_j defined by Vi_j, r {x) = ^(rXi-j) 
for allx G U itj . Then, &,_,> = (u it _j tr ) Um ® . 



Proof. For simplicity, we write H = £/»,_,, K = U[ a, ft = IM,-j,n v = Vi,-j, r an d 
( = (i,-j,r- It is clear that v is a linear character of K. By Frobenius' reciprocity, 
we have (£,v Um ( q >) = ((k>v), whereas, by Mackey's Subgroup Theorem (see [T7J Theo- 
rem 17.4(a)]), 



CK = (^ {q) )K = J2^H^n K ) 



PxHx-^K) 
xeX 



where X C U m (q) is a complete set of representatives of the (H,K)-doub\e classes of 
U m (q); without loss of generality, we choose X so that 1 & X. Thus, we obtain 

(Cv Um(q) ) = J2^h x -^k) R » = E^U^n*. 

x£X x£X 

In particular, for x = 1, we get (/i, u) HnK = 1 (because, both \i and v are linear), and thus 
((, i/ Um ^) ^ 0. Since ( is irreducible (by [151 Corollary 5.11]; see also [21 Lemma 3]), we 
conclude that ( is an irreducible constituent of v Um ( g \ Since \U m (q) : K\ = \U m (q) : H\, 
we obtain £ = u Um ^ as required. □ 

We observe that, if a = Si + £,- for 1 < z < j < n, then 

U^ HU = U a and ^.t^ = U m (q) . 

As a consequence of this and the above observations, we may prove the following result. 

Proposition 3.2. Let a G $, Zet (i,j) £ £ + (a), and suppose that j ^ —i (in the case 
where U < Sp 2n {q))- Then, (Ci,j,r)u = (£-j,-i,r)u = Z,a,r for all r e F, 



X 

9 ' 



Proof. For simplicity, we set ( = Q,j,r- Let -K" = U it j if j ^ 0, and K = U-j if -< j -< — i. 
As we observed above, K C\U — U a and iff/ = U m (q). On the other hand, let /i = /ijj ir 
if j ^ 0, /i = z/jj if -< j -< — i. By Mackey's Subgroup Theorem, we get (fi Um ^)u = 
(l^Knu) U ■ Since K nU = U a and /x^nt/ = \*,r> we conclude that (u = £ a>r , as required. 
The proof of the equality (C-j,-«,t-)i/ = (,a,r is analogous. □ 

The previous lemma is not true in the case where U < Sp2 n (q) and a = 2Ei for 
1 < i < n. In fact, we have U a < U-_ { (hence, U[_ i U OL = U-_ { ^ U m (q) whenever 
% > 2). In order to deal with these cases, we start by proving the following auxiliary 
result. (The subsets K Vip C U m (q) are exactly the superclasses of U m (q) as explained in 
[TBI Appendix A]; see also the papers [U HJJ, or the PhD thesis [2"5].) 

Lemma 3.3. Let V be a basic subset of S, let (p: T> — > F x fre a map, and let 

ev,<p= ^ if(i,j)eij e Um(q). 



10 



Let Ot> }V , = U m (q)e V)ip U m (q) C u m (q) and K Vjtp = 1 + O v>(p C C/ m (g). On tfie oi/ier /land, 

(X Xf iw \ /u V w \ 

J r -f T J G C7 and a z = I r -w T J G u 
Jar T J/ \0 -Ju T J) 

where x = 1 + u. Then, z G ^-d,^ if and only if a z G Ox> >v - Moreover, the mapping 
z *—>■ a z defines a bijection from U to u. 

Proof. We only consider the case U < Sp2 n {q) (the others are similar); hence r = 0. 
Since 

x 0\ (x w \ (x~ x N 

l) \0 Jx- T j) ^ l y 

and Kj) t(p is invariant under ?7 m (g)-conjugation, we conclude that 

z e k v . v «=► ^ Jx _ Tj j g i^. 

Since x _1 — 1 = — mx _1 , we have Jx~ T J — 1 = (Jx~ T J)(—Ju T J) and so 

u w \ _ / 1 \ Az w 

Jx- r J-lJ~\0 Jar T jHo -J« T J 



It follows that 



X t/7 



and this completes the proof. □ 

We are now able to prove the following result. Given any W q - vector space V and 
any linear map / G V*, we denote by df the composite map $ o /: V — ► C x ; it is 
straightforward to check that df is a linear character of the additive group V + and that 
Irr(V + ) = {tf /: feV*}. 

Lemma 3.4. Suppose that U < Sp2n(q), let a = 2ei for 1 < i < n, and let r G F x . 
Then, ^ r is an irreducible constituent of (Ci,-i,r)u w ^h multiplicity 1; in particular, 

£,a,r T 1 \{yi,—i,r)U- 

Proof. For simplicity, we write £ = £ Qjr and ( = Ci,-i,r- We evaluate the Frobenius' 
product (CutO- Since £ = X u where A = X a>r , we have ((u,0 = (Cu a ^)- Let u 2n (g)* be 
the dual space of u 2n (g), let e\_ { G u 2n (g)* be defined by e*_j(a) = a^ _j for all a G u 2n (g), 
and let O C u 2n (g)* be the coadjoint ?7 2n (g)-orbit which contains re*_j. Then, by [T5] 
Corollary 5.11], we have 

V \ U \ feo 
for all a G u 2n (g); in fact, O = U 2 n{q){rel_ i )U 2 n{q)- 



11 



Let z G U a be arbitrary, and let a z G u be the element defined in the previous lemma; 
it is clear that a z G u a . By [3], Theorem 1], there exists a (unique) basic subset DCS 
and a (unique) map tp; T> —* F x such that z G K?>, v - By the previous lemma, we have 
1 + a z G i^D,^, and so 

c(z) = c(i + ^) = -^=E^(^)- 

V\ u \ feo 
Since the mapping z i— > a 2 defines a bijection from C/ a to u a , we conclude that 



/GO V 1 a| a€u a / V 1^1 /gO 



and thus 



,, u l{/GO:/ -re*_,G(u a m 
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where (i^)- 1 = {/i G u 2 „(g)*: h(u a ) = 0}. 

Now, we consider the basis {ep : /3 G $} of u. Let \l/ = $ — {e^ — e^ : z < fe < n}, so that 
{e/3 : (3 G \l/} is a basis for u a . By [3j Lemma 1], in order to describe Or^Uc,)- 1 , it is enough 
to consider vectors ep for (3 G \P satisfying (j, fc) G £(/3) for some i ^ j -< k -< —i. Firstly, 
suppose that (3 = Ej — £& for i < j < k < n. In this case, we have ep = e^ — e-k-j, and 
so h(ej t k) = h(e_k-j) for all h G (Uq,) -1 . Similarly, if (3 = Sj + e fc for i < j < k < n, then 
ep = ej-k + ek-j, and thus h(ej-k) = —h(ek,-j) for all h G (Ua) 1 . Finally, if /3 — 2ej for 
i < i < n, then eg = e^- -j, and so h(ej-j) = for all ft, G (Uq) 1 . Now, suppose that h = 
f - rel_i G (u a ) x for / G O. Then, since f{e i - i ) = r _1 /(*, -J)f(ej-i) = -r^fi^-j) 2 
(see jH Lemma 1]), we deduce that /(e^-y) = — /(e^-i) = f{^j-j) = for all i < j < n, 
and this clearly implies that 

Since yfO] = C(l) = g 2(n ~ 4 - ) , we conclude that (£c/,£) = 1 as required. D 

A similar argument can be used to prove the following result. 



x 
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Lemma 3.5. Suppose that U < Sp2 n (q), let a = 2£j /or 1 < z < n ; and let r G F 
Lei £ = Ci,-i,r ^ e ^ e supercharacter ofU2n(q) associated with (i, —i) and r. Moreover, let 
(3 = 2s j for i < j < n, and let s G F x be such that —rs G (F x ) 2 . Then, the (irreducible) 
supercharacter £ = £ a ,r£/3,s is a constituent of (u with multiplicity 2. 

Proof. By the definition, we have £ = X u where A = \ a ,r^fi,s is the linear character of 
V = U a nUp defined by X(x) = ~&(rxi-i + sxj-j) for all x G V. Thus, ((u,0u = (£v> A) v. 
For each z G £7, let a 2 6 u be as in Lemma l3~3l Then, for h = re* t + se*_j G U2 n (o)*, 
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we have X(x) = ^(a^) for all x E V . Therefore, since x h-> a x defines a bijection from V 
to o = u a fl Up, we deduce that 

where O = U 2n (q)(re*_ i )U2n(q) ■ Thus, we get 



(&, a) 
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/ "(F x ) 2 , and thus —rs E (F g x ) 2 ; moreover, /(e^-j) must be non-zero. Therefore, we 



Now, let * = $ — {£j — e k ,Ej — E[-. i < k < n, j < I < n}, so that {e^ : (3 G ^} is a 
basis for t>. As in the previous proof, for an arbitrary element g 6 D 1 , we obtain: 

(i) g(e kt i) = g(e-i- k ) for all (3 = e k - e x G * with i < k < I < n; 
(ii) g(e k -i) = -g(ei- k ) for all /9 = e fc + ^ G ^ with i < k < I < n; 
(iii) g(e k - k ) = for all /3 = 2^ G \l/ with i < k < n. 

Suppose that / — h G O 1 - for some f E O. Then, since f(e k - k ) = r~ x f{i, —k)f(e k -i) = 
-r~ l f(ei _ fe ) 2 , we deduce that /(e* - k ) = -f(e k -i) = f{e k - k ) = whenever % -< k <n 
and k ^ j. On the other hand, we have s = h(ej -j) = f(ej-j) = —r~ 1 f(ei-j) 2 E 

conclude that 

\{fEO:f-hEt, ± }\ = 2q 2 ^\ 

and this completes the proof. D 

In the notation of the two previous lemmas, we deduce that 

%U — £,a,r + 2 2_^ 2-^1 £,a,r£,2sj,s + V 

i<j<n sC-r- 1 (¥ q x ) 2 

where 77 is, either the zero function, or a character of U. Taking degrees, we obtain 

q —± q ^\ + r 1 {l) = q^+r 1 {l). 

i<j<n / 

It follows that 77(1) = 0, and this concludes the proof of the following result. 

Proposition 3.6. Suppose that U < Sp2 n {<i), let a = 2Ei for 1 < % < n, and let r E F x . 

Let ( = Q,-i,r be the supercharacter ofU2 n (<l) associated with (i, —i) and r. Then, 

Cu = £,a,r + 2 2_^ 2-^1 £a,r&ej,s- 

i<j<n se _ r -i( Fc3 x )2 
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4. The algebra of superclass functions 

The main goal of this section is to prove the existence part of Theorem 12.11 We start 
by proving a result on the decomposition of the product of two elementary characters of 
U. The argument of the proof uses the corresponding decompositions in the case of the 
unitriangular group U m (q), which can be found in J2j Lemmas 6-8 and 11]. (The proofs 
in that paper use the assumption p > m, but can be easily adapted for an arbitrary 
prime (see the thesis [22j by N. Yan); in fact, every irreducible constituent of any of the 
given decompositions is a supercharacter of U m (q), and thus is a Kirillov function (see 
[151 Theorem 5.10], or [6, Theorem 3].) 

Proposition 4.1. Let «,/?€ $, and r,s G F g x . Then, the product £a,r6/3,s decomposes as 
a sum of supercharacters . 

Proof. The result is obvious in the case where {a, (3} is a basic subset of $. Thus, we 
assume that {«,/?} is not basic. Let (i,j) G £ + (ot) and (k, I) G £ + {(3); without loss of 
generality, we may assume that % < k. We observe that, by the definition of basic subset 
of roots, £({a, /3}) = {{i,j), (—j, —i), (k, I), (— /, — k)} is not a basic subset of £. 

Firstly, we assume that j ^ —i and / ^ —k (in the case where U = Sp2n(q))- Suppose 
that {(i, j), {k,l)} is a basic subset of £; hence, we must have k = —j. By the previous 
lemma, we have £ a , r £p, s = (Ci,j,r)u(C-j,i,s)u = ((i,j,r)u((-i,j, s )u, and thus, by [21 Lemma 7], 
we obtain 

^a,r^/3,s = (Ci,j,r)u + /_^ Z_^ (Ci,j,r)u(C-l,b,t)u = £a,r + 2_^ 2_^ £a,r((-l,b,t)u- 

-Kb<j teFg x -Kb<j teFg x 

The result follows by the Propositions 13.21 and 13.61 

On the other hand, suppose that {(i, j), {k,l)} is not a basic subset of £. Firstly, we 
consider the case where i = k and j > I. By [21 Lemma 6], we obtain 

£a,r£,/3,s = £,a,r + 2_^ Z_^ £ot,r\(a,l,tJU, 

i^a^l ieFq X 

and the result follows because the subsets {(i, j), (a, /)}, for i -< a -< /, are all basic. The 
case where i < k and j — I is similar, because 

Za,r<,(3,s — £a,r + 2_^ Z_^ £a,r\Ck,b,t)U 

k~ib<j teF(j x 

(by [21 Lemma 6]), and the subsets {(z, j), (fc, &)}, for k -< b -< j, are all basic. 

Finally, suppose that i = k and j = /; hence, a = (3. On the one hand, if s ^ — r, then 

fa.rfa.s = (1 + (q - l)(j - i ~ l))(Ci,j,r+s)u + Yl 5Z ^ ~ l )((i,j,r+s)u((a,b,t)u 
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(by [21 Lemma 11]), and the result follows as in the previous cases. On the other hand, 
suppose that s = — r. In this case, [21 Lemma 8] implies that 

Z,a,rt,a,-r = 1(7 + 2_^ / j (Ca,j,t)u + / , / „ (d,b,t')u + / , / J {(a,j,t)u{(i,b,t')u , 
«-(a^j tGF<j X j-<6^j f G F g x i-<a,b^j t,t'e¥ q x 

and thus the result follows by the same reason. 

Next, we assume that U = Sp2 n {.<l) and I = —k. Since {a,/3} is not basic, the subset 
{(i, j), {k, —k)} cannot be basic; we recall that £({a,j3}) = {(i,j), (—j, —i), (k, —k)}. We 
start by considering the case where i ^ k and j = —k. By [21 Lemma 6], we obtain 

€a,r(Ck,-k,s)u = £a,r + /_^ /_^ £,a,r(Ck,b,t)u- 

k<b<-k t& * 

Since £, a ,r£,p,s is a constituent of £ a , r {Ck,-k,s)u (by Lemma E3J), it is a sum of some of 
the irreducible constituents of the characters (, a ^ r and £, a , r ((k,b,t)u for k -< b -< —A; and 
t G F g x ; we observe that £ a ,r(Cfc,6,t)i/ = id -k,r)u{(k,b,t)u = {(k,-i,r)u{(k,b,t)u is reducible 
(in general). Let k -< b -< —k and t G F x be arbitrary. Then, by [21 Lemma 6], we obtain 

£,a,r((k,b,t)u — (Ck,~i,r)u(Ck,b,t)u = £,a,r + 2__, Z-C £<v(Ca,6,£')y- 

fe^o^6t' e F ? x 

Using Propositions I3.2l and l3.6l it is now easy to conclude that the irreducible constituents 
°f £a,r(Ck,b,t)u are supercharacters of U, and so the result also follows in this situation. 

Now, suppose that (i,j) = (k, —k) = (i, — i)\ hence, a — (3. On the one hand, suppose 
that s = —r. Then, by [21 Lemma 8], ((i-i t r)u((i,-i,-r)u decomposes as a sum with 
constituents of the form (( a ,-i,t)u((i,b,t')u = ((i,a,t)u((i,b,t>)u for % -< a, b -< -i and t, t' G ¥ q ; 
for simplicity, we set C«,v,o — ^u m (q) for all (u,v) G £. By the above, each character 
((i,a,t)u((i,b,t')u decomposes as a sum of irreducible supercharacters, and thus £ a ,r£,a,-r 
also decomposes as a sum of irreducible supercharacters (because £ a ,r£a,-r is a constituent 

of (Ci,-i,r)u(Ci-i-r)u)- 

On the other hand, suppose that s ^ —r. Then, by [21 Lemma 11], (Ci,-%,r)u{Ci,-i,s)u 
decomposes as a sum with constituents of the form (Ci,-%, r +s)u(Ca,b,t)u for 2 -< a -< 5 ^ — i 
and t G ¥ q . Let i -< a -< b -< — i and t G F g x ; we observe that, by Proposition I3.2[ 
we may assume that 1 < a < n and a -< b -< —a. By Proposition 13.61 the irreducible 
constituents of (Ci,-i,r+s)u(Ca,b,t)u are also irreducible constituents of characters with the 
form £a,r(Cc,-c,u)u((a,b,t)u for some i -< c ^ n and some u G ¥ q . Using a simple inductive 
argument, we may assume that each character (Cc-c,u)u((a,b,t)u decomposes as a sum 
of irreducible supercharacters, and this clearly implies that £ a ,r{Cc,-c,u)u{Ca,b,t)u also de- 
composes as a sum of irreducible supercharacters. The result follows because £, a) r£,a,s is a 
constituent of (Ci,-i,r)u((i,-i,s)u- 
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Finally, we assume that i = k and j -< —k = —i (we recall that i -< j ;< — i). In this 
case, [21 Lemma 6] implies that 

£a,r(Ci ,-i,s)u = (G -i,s)lj + 2_^ / J ((i,-i,s)u((a,j,t)u- 

As in the previous case, since £ a ,r£,/3,s is a constituent of £ a ,r((i,-i,s)u, it is a sum of 
some of the irreducible constituents of the characters (Ci,-i, s )u an d (Ci,-i,s)u(Ca,j,t)u f° r 
i -< a -< j and £ G F g x . Using Propositions 13.21 and I3.6[ we easily conclude that the 
irreducible constituents of (Ci,-i, s )u an d (Ci,-i, s )u(Ca,j,t)u, for z -< a -< j, a 7^ — j, and 
£ G F x , are supercharacters of £/. It remains to consider the irreducible constituents 
of (Ci-i,s)u(C-j,j,t)u f° r t G F g x . However, by Proposition 13.61 and by one of the cases 
considered previously, all these irreducible constituents are supercharacters. 

The proof is complete. □ 

Next, we prove that the product of supercharacters is a linear combination (with 
nonnegative integer coefficients) of supercharacters. For the (inductive) proof, we need 
to endow the set of roots with the total order -< defined as follows. Given a, j3 G 3>, we 
choose (i,j) G £ + (a) and (k, I) G £ + (/?), and set a -< /3 if and only if, either / -< j, or 
j = I and i -< j. The following observation will also be very useful (and is an immediate 
consequence of the previous proof) . 

Lemma 4.2. Let a, (5 G $ be roots with a -< j3, and let r,s G F g x . Let D C $ be a basic 
subset, and suppose that the supercharacter £0,0 is a constituent o/^ a ,r^/3,s for some map 
cf): D — > F x . Let 7 G D be the smallest root in D. Then, a -< 7, and a 7^ 7 if and only 
if a = (3 and s = —r. 

We may now proceed with the proof of the following result. Here, we denote by cf (U) 
the unitary vector space consisting of all class functions of U, and by scf(U) the vector 
subspace of cf(£7) spanned by all supercharacters of U. ("scf" stands for "superclass 
function"; superclasses of U will be defined in the forthcoming paper [S].) 

Theorem 4.3. The product of two supercharacters of U decomposes as a sum of super- 
characters. In other words, scf([7) is a subalgebra ofci(U); moreover, scf(£7) is finitely 
generated (as an algebra) by the elementary characters of U . 

Proof. Let D,D' C $ be nonempty basic subsets, and let £0^ and £d',4>' be superchar- 
acters of U associated with maps 0: D — > F x and <$' : D' — > F x . Let a G D', and 
D' = D' — {a}. By the definition, we have £d',^' = ia,r£,D' ,<p' where r = <p'(a) and (p' 
is the restriction of 0' to D' , and thus we may use induction on \D'\ to conclude that 
£d' ,<i>'£,d,4> decomposes as a sum of supercharacters. Therefore, we are reduced to the case 
where £d',<// = £ Q , r ; in other words, we must prove that the product ^ a ,r^D,cj> decomposes 
as a sum of supercharacters. To see this, we will proceed by reverse induction on the 
set of all basic subsets of $ endowed with the lexicographic order ■< which is naturally 
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determined by the total order ^ on $ (as defined above). (We observe that the maximal 
basic subset of $ (with respect to this order) is {^i — £2}-) 

Let D and <fi be as above. Moreover, let a G <& and r G F x be arbitrary, and consider 
the product £, a ,r£,D,<i>- Let (3 G D be the smallest root in D, and let s = 0(/3) G F x . 
If .D = {/?}, then £ Q)r £D,0 = £,a,r£,p,s decomposes as a product of supercharacters (by 
Proposition ^. II) . Thus, we assume that \D\ > 2. Let D = D — {j3}, and let 0o : -Do — ► F g x 
be the restriction of <p to Dq. 

Firstly, suppose that a -< (3. Since D -< D , the product £ a ,T-£D ,<fo decomposes as a 
sum of supercharacters (by reverse induction). Let D" C $ be a basic subset such that 
£d",4>" is a constituent of £ a ,r£Do,#o f° r some map 0": D" — > F g x . Since a $. D (by the 
minimal choice of /?), a is strictly smaller than all the roots in D Q . Therefore, Lemma H~2l 
implies that a G D", and soD^ £)". By reverse induction, we conclude that the product 
£,{3,s£,D" ,<t>" decomposes as a sum of supercharacters, and thus £ a ,r£,D,<f> = £p,4>(p)(£ a ,r£D ,(f>o) 
also decomposes as a sum of supercharacters. 

On the other hand, suppose that (3 -< a. As in the previous case, the product ^ Q ,rCi?o,?io 
decomposes as a sum of supercharacters, and so we may choose a basic subset D" C $ 
such that ^d",0" is a constituent of £, a , r £,D ,<i> for some map <fi" ': D" — >■ F g x . By Lemma 
I4.2[ we see that the smallest root in Dq U {a} is smaller or equal to the smallest root in 
D", and so (3 is strictly smaller than the smallest root in D", which means that D -< D" . 
Thus, as in the previous case, we conclude that the product £,p >s £d",4>" decomposes as a 
sum of supercharacters, and thus £ a ,r£D,<f> — £/3,<j>(/3)(£a,r£D ,4>o) a lso decomposes as a sum 
of supercharacters. 

Finally, suppose that (3 = a. In this case, we have £, a ,r£,D,<t> = {£a,r£a,s)£D ,(j>o- By 
Proposition 14. II and Lemma H~2| £, a ,r£,a,s decomposes as a product of supercharacters with 
the form £ a ,t£D",ct>" where D" C $ is a basic subset such that a is strictly smaller than 
all of its roots, <ft" : D" — > F g x is a map, and t G F g ; for simplicity, we write £ Q) o = If/. By 
successively repeating the arguments above, we deduce that £d»,, ? j»£do,0o decomposes as a 
sum of supercharacters, each one corresponding to a basic subset with smallest root larger 
than a. Therefore, D is smaller than all of these basic subsets, and so we may use reverse 
induction to conclude that (, a ,t£,D",<i>"£,Do,<f>o & iso decomposes as a sum of supercharacters. 
It follows that £ a ,r£D,</> decomposes as a sum of supercharacters, and this completes the 
proof. □ 

As a corollary, we obtain the following result. 

Corollary 4.4. The restriction Qu of any supercharacter ( of U m (q) decomposes as a 
sum of supercharacters ofU. 

Proof. Since ( is a product of elementary characters of U m (q), Propositions 13.21 and 13.61 
imply that (u is a product of elementary characters of U, and the result follows by the 
previous theorem. □ 

We are now able to prove the existence part of Theorem 12.11 
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Theorem 4.5. Every irreducible character x of U is a constituent of a supercharacter of 
U; in other words, there exists a basic subset D of $ and a map (f>: D — > F g x such that 

Proof. Let ip be an irreducible character of U m (q) with (x,i>u) 7^ 0. Let PC ^ be the 
(unique) basic subset and <p : V — > F x the (unique) map such that ?/> is a constituent of 
the supercharacter ( = £p j¥ , of U m (q). Then, ipu is a constituent of (u, hence x is an 
irreducible constituent of Cc/- The result follows by the previous corollary. □ 

5. Orthogonality of supercharacters 

In this section, we prove the orthogonality of supercharacters, and thus complete the 
proof of Theorem 12.11 The proof depends on the decomposition of u* as a disjoint union 
of its "basic subvarieties" as defined in the authors' paper [7] . We start by recalling their 
definition. 

We fix an arbitrary non-empty basic subset D C $, and define the D-singular and 
.D-regular entries as follows. For any (i,j) G £, we set 

S(i,j) = {(i, k)e£:k^j}U {(k,j) e8:i^k}, 
and define, for any a G $, the subsets 

£s(a) = U 3(i,j) and E R (a) = E - £ s (a) 

of E. We say that an entry (k,l) G £ is a-singular if (k,l) G Es(a); otherwise, we say 
that (k, I) is an a-regular. More generally, given an arbitrary basic subset T> C E, we 
define 

£(£>)= |J 5(i,j) and R(V) = £ - S(V). 

The entries in S(T>) are said to be T>-singular, and the entries in R(T>) are said to be 
V-regular. We observe that, for any a G $, an entry (fc, /) G £ is a-singular (resp., 
a-regular) if and only if it is £(a)-singular (resp., £(a)-regular). Now, since D C $ is 
a basic subset, £(£*) = {J a( zD^( a ) ^ s a basic subset of £ (by the definition). We say 
that an entry (i,j) G £ is D-singular (resp., D-regular) if (i, j) is £(.D)-singular (resp., 
£(.D)-regular). We denote by £s(D) the subset of £ consisting of all D-singular entries, 
and by £r(D) the subset of £ consisting of all D-regular entries. It is clear that 

£ S (D) = |J £ s (a) and £ R (D) =£- £ S (D). 

(This definition can be extended to the empty (basic) subset of $, in which case all entries 
in £ are regular.) 
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For an arbitrary basic subset T> C £ and an arbitrary entry (i,j) G S, we denote by 
V(i,j) the subset 

V(i,j) = {{k,l)eV: l^k^i, j^l^-l}; 

it is clear that V(i,j) is a basic subset of £. Let T>(i,j) = {(ii,ji), ■ ■ ■ , (it,jt)} and 
suppose that j\ -< • • • -< j t . Moreover, let a G St be the (unique) permutation such that 
Mi) ■<•••-< icr{t) j as usual, we denote by S t the symmetric group of degree t. Then, for 
any / G u m (q)*, we define Afj(f) G ¥ q to be the determinant 



(5d) 



< ? (/) 









We note that, if X>(i, j) is empty, then Afj(f) = /(ejj); in particular, if D is empty, then 

A5(/)=/(ey)faraU(i,j)e5. 

Now, for any / G u*, we define the element «(/) G u by 



«(/) 



E 



u(f) a e c 



where, for each a G $, we set 

{I f(e a ), if a = £j ± £j for 1 < 2 < j < n, 
/(e Q ), if u < sp 2ri (<?) and a = 2s, for 1 < % < n, 
\ f(e a ), if u < 2n +i(g) and a = e t for 1 < i < n. 

It is easy to see that f{v) = Ti(u(f) T v) for all v£u, and that the mapping / i— > u(f) 
defines a vector space isomorphism from u* to u. Finally, we define / G u m (g)* by 

f(v) = Tr(u(f) T v) 
for all v G u m (q). Then, for any basic subset D C $ and any entry (2, j) G £, we set 

AS(/) = Ag D) (/) 

for all / G u, and, for any map : D — > F^ x , we define the fraszc subvariety 

D ( p\ x D 



0* Dt4 , = {/eu': A&(/) = A&(/ A ,) for all (i,j) G £*(£>)} 



where 
(5e) 



/. 



D,. 



Z>(«) 



el eu* 



ie-D 



The following result is Theorem 4.5 of the paper [7] (the proof given there is valid for 
an arbitrary odd prime). 
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Theorem 5.1. For any basic subset D C $ and any map 0: D — > F 9 X ; £/ie 6aszc sufc- 
variety 0* D ^ C u* zs XJ -invariant (for the usual coadjoint action). Moreover, the vector 
space u* decomposes as the disjoint union 



u* = {JO 



D, 



of all its basic subvarieties. 



As a particular case, let a G $ and r G F x . Then, for D = {a} and 0: D — > F 
defined by 0(a) = r, we obtain the elementary subvariety 



(1* — (1* 
By Theorem 5.5], we have 

( 5f ) °D^ = E °M(a) 

for any basic subset D C $ and any map : D — > F g x . The elementary subvarieties of u* 
determine the elementary supercharacters of C/ (and vice- versa) as shown by the formula 
of Corollary 14.41 below. The following observation will be useful for the proof. 

Let D be a basic subset of $, and let 0: D — ► F x be a map. Let /d^ G u* be defined 

as in (l5el) . and write /p^ = forf G u m (g)*. By the definition, for any / G u*, 

/ G ^ <=► Ag D) (/) = Ag D) (/ D ^) for all (i,j) G £ H (I>). 

Henceforth, we denote by v?d the map ipa'- £(D) — > F g x defined by <p D (i,j) = fn^i^ij) 
for all (i,j) eS(D). Let 

°£(d),vd = U ™(q)fD,<t>U m (q) C u m (g)*. 
By [21 Propositions 1 and 2] (see also the discussion in [151 Appendix 2]), we know that 
(5g) 0* e(D)i(pD = {fe u m (qT ■■ AJ(/) = A^(/^) for all (*, j) G £«(£>)}, 

and thus 

0* D ^ = {feu*:feOl {D) ^ D }. 

In particular, let a £ Q, (i,j) G £ + (a) and r G F g x be arbitrary. Then, for D = {a} 
and 0: -D — > F g x defined by 0(a) = r, we have /d^ = re* and /^ = (r/2)(e*.- ± e* -_J 
(where the sign is well-determined by u). By J3j Proposition 1 and 2], we know that 

°5(a), Va = °tj,r/2 + °-j,-i,±r/2 where <P<* = ^D = V{a}, and thus 

(5h) 0; r = {/ G u* : / G 0* iir/2 + OV _ ii±r/2 }. 

We are now able to proceed with the proof of the following result. 
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Proposition 5.2. Let a G $ and let r G F g x . For any z & U, we denote by a z the 
element of u given by Lemma EOl Then, 



Ca,r{z) = 1#T E */(* 



10* 

1 Q ' rl /eo S . r 



/or a// z <E U. 



Proof. Let (z, j) G £" + (a:). Firstly, we consider the case where j ^ —i. By Proposition 
E£H we have £ Q , r = (Ci,j,r)t/- Let 0* jr = U m (q)(re* j )U m (q) C u m (g)*. Then, by Lemma 
13.31 and by [15, Corollary 5.11], we deduce that 

£«,r(z) = CiJ,r(«) = CiJ,r(l + «*) = i ^2 ^/( W *) 

y l°*j,rl /eOf ii>r 

for all z £ U. 

Let / G 0* jr be arbitrary, and consider the restriction / u of / to u. Define w(/ u ) G u 

and / = / u G u m (g)* as above; hence, /(u) = Tr(-u(/ u ) T t>) for all v G u m (g). Let 
(^: £(a) — » F g x be the map defined by ip(i,j) = u(f)ij = u(f) a = r/2 and f(—j, —i) = 
u(f)-j-i = ±r/2. Using ( |5g| ), it is straightforward to check that / G O* • r / 2 + 0* • i ±r , 2 , 
hence f u G 0* ar (by fl5h|) ). Since the mapping / i— > / u clearly defines an injective map 
from 0* Jjr to u and since |0* r | = |0* - r | (by direct computation), we conclude that 

o; r = {/u:/6o;, r }. 



a,r I 



The result follows because £ a , r (l) = \ \0% 

On the other hand, suppose that U < Sp 2n (q) and a = 2£j for some 1 < i < n. In this 
case, by [H Proposition 3.1 and Theorem 5.5], O* r C u* is the coadjoint ?7-orbit which 
contains re* . Let 2 G [7 be fixed. By the definition of induced character, we have 



Za,r{Z) — (A a ,r) iz) — . . / j \ a ^[xZX 



xeu 

XZX~ 1 £U a 



Since X a ^ r (xzx x ) = \ a ^ r (xa z x x ) for all x G £/ with x^x x G t/ a , we deduce that 

£o,r(*) = 777-7 JZ ^rej (^aT 1 ) 



\Ua\ 

xeu 

XZX~ 1 GU a 



Y^^2^re* a (xa z X X ) — ^ ^g{xa z X X ) 



\TT I / j « c ft V ^ / I I 

9G(u a ) 
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1 ' geiuc) 1 - x£U ' ' feere*+(u Q )- L xGf/ 

Now, it is straightforward to check that re* + (Uq) 1 C O* r , and so 

for all /i G re* +(u Q )- L ; as usual, Cjj{re* a ) denotes the centralizer of re* under the coadjoint 

action of U. It follows that 

t f s \Cu{re* a )\\U: U a \ ^ \U • U a \ 

Ur{z) = [77] 2^ § ^ = \n* I 2_, ^/( a *)- 

The result follows because £ a ,r(l) = |£/ : £7 a | (by the definition). D 

Remark 5.3. We observe that, by [TJ Proposition 2.1], a basic subvariety O* r , for a G $ 
and r G F g x , is a coadjoint [/-orbit in all cases, except when U is orthogonal and a = Si+Ej 
for some 1 < i,j < n. In this case, by [7, Theorem 5.5], we have 

0* a , r = [jo s 

s& q 

where (3 = £j — £j and O s = 0{re* a + se^) denotes the coadjoint [/-orbit which contains 
re* + se*n G u*. On the other hand, an argument similar to the above shows that, for any 
sGF„ the expression 

Xs(z) = ^ J2 ^ a *)> for a11 z e U ' 

defines an irreducible character of U. In fact, we may consider the subgroup 

V = {z G U: a z G u a + F^} 

of U, and define the linear character /x s : V — > C x by /i s (z) = "&(rzi t -j + szij) for all 
z & V. Then, we may show that Xs = (a*s) for all s G ¥ q , and that 

<,a,r / , Xs 

s& q 

is the decomposition of ^ a<r into q distinct irreducible constituents (cf. Proposi- 
tion 2.1]). 

Next, we prove the orthogonality of supercharacters (and thus we conclude the proof 
of Theorem 12.11) . 



Theorem 5.4. Let D and D' be basic subsets o/$ and let <p: D — > F^ x and 0': /}'—>• F g x 
6e maps. Taen, (£ Dj(/> , £ D ',<t>') t^OjJ and on/y if (D, 4>) = (£>', 0'). 
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Proof. Let z G U be arbitrary, we let a z G u be the element given by Lemma 13.31 By the 
definition, we have ^r>^ = YiaeD^a,4>(a), an d so (by the previous proposition) 

**>*& = I^T E ^(^) • • •*/>«) = %^T E m ^ (a ^ 

where d= |D|, O^ = FLes °« ><Ka) , and 

m/ = |{(/i,...,/ d )enV/i + "- + /- = /}l 

for all / G 0* D<p ; we recall that 0* Dlp = J2 aeD 0* a ^ (a) (see ©). Similarly, 

■■ - la. 



Zd>A z ) = t^ r 2^ m /. 



10* 



SV"-^ 



where fi*,,^ = Ylp GD >°h(py and 

K = l((#i> •••>&*') G ^*D',4>'- 9i + --- + 9d> = g}\ 

for all g G 0* DI( u] here, d' = |D'|. Since the mapping z \— > a z defines a bijection U — > u, 
we deduce that 



10* I 10 



^r E E (4£*/(*)W) 



WW WW ^ ^ ( ^ /A)u 



10* I 10* 

£d,0(1)£d',0'(1) 



d.*m d> /eO^seO* 



| * I i * I \ w D,<p l ' °V,<//b 
I ".0,0 1 l"l)',0'l 

and the result follows by [7J Theorem 4.5]. D 

6. A SUPERCHARACTER FORMULA 

In this section, we establish a formula for the values of a supercharacter ^^ as a sum 
over the the basic subvariety 0* D , C u* (which extends Proposition 15. 2p . In fact, we have 
the following result. 

Theorem 6.1. Let D be a basic subset of $, and let <fi: D — *■ F g x fre a map. For any 
z & U, we denote by a z the element of u given fry Lemma \3.3[ Then, 

&,*(*) = 77#r E M**) 

for all z G U . 
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Proof. Let z G U be arbitrary. As in the previous proof, we have 
where fi^ = fLeD °a,,«a)> and 

aG-D 

for all / G 0^. 

Now, let Z> = £ + (.D), let ip: V — > F g x be the map defined by <p(i,j) = 0(a) for all 
(i,j) G V with (i, j) G 5(a), and consider the supercharacter (x>,<p of U m (q). By Lemma 
13.31 and [T5l Theorem 5.6], we have 



&M = &,*(! + a») = fe^T E */ 






Let 7r: u m (g)* — ► u* be the natural projection (given by restriction of functions). Since i\ 
clearly defines an injective map 7r: O^ — > u*, we obtain 






f{a z ) 



V,tp\ 



f^(o* Vip ) 



It is straightforward to check that 0* D ^ C vt u (0^, j(/:j ); in fact, we have 7r(0?- ) C O* r for 
all a G $, all (i, j) G 5(a) and all r G F g x (the equality holds whenever j ^ —i; see 
the proof of Proposition I5.2|) . and the claim follows because 0* D i = E a eD^«<4(a) (°y 
[3 Theorem 5.5]). Since tt(0£, ) and O^, are [/-invariant, we conclude that 7r u (0|> ) 
decomposes as the disjoint union 







vr u (0. 


D,tp) — 


(1* 


uv 


some 


[/-invariant subset of V C u*. 


Therefore, 


we get 




d>, v (z) = 


_ Wl) I 
\Ot> 1 1 


£ 


Mo 


'**) + E^^ 2 






\^V,ip\ \ 


\fe°h„ 


p 


feV 



On the other hand, by Proposition 14.11 and Theorem 12.11 we know that 

(Cv,<p)u = m D ,<i>£D,4> + V 

where rj is a linear combination (with nonnegative integer coefficients) of supercharacters 
satisfying (£d,</» v) = 0- Arguing as in the proof of Theorem 15.11 we obtain 

feV 
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for some subset V C u* and some positive integers nf for / G V . Therefore, we get 
Cv, v (z) = rriD^D^z) + r](z) = ^ D 't — Yl m f^f( a ^ + Yl n f^f( a ^- 

I D,4> I fe0 * D4i /gV 

Since 2; E [/ is arbitrary and the mapping z >— > a z defines a bijection, we conclude that 

fe - (1) 1 e w^wU y E ™aw + E"AW 



in* I l A^ J K ' Z-^i J v ' l in 

for all a G u, hence 



v ' vl WeO*, /ev / ' A*< /eo* /gV 



%^H E '/ + £*/] - ""ffil' 1 ' E '«a + E"A 

(as linear characters of the abelian group u + ). Since 0* D ^C\V = 0* D ^r\V' = 0, we deduce 
that 

Cxy(l) = m D ^ D ^(l)m f 
\(1* I ~ 10* I 

for all / G 0* D ^. Therefore, the coefficients rrif do not depend on / G 0* D ^, and thus 

£ D Al)m 






for a well-determined positive integer ra. Taking degrees, we obtain m = |^d J/|0£, J, 
and so 

as required. D 

As an immediate consequence, we obtain the following result. 

Corollary 6.2. Let D C Q be a basic subset, and let <p: D — »■ F g x be a map. Then, 
{Zdj, Zdj) = Zd,j>(1) 2 /\0* d J; hence, \0* D J = £d,^(1)7(W> W>- 

Proof. Using the formula of the previous theorem, we evaluate 

as required. D 

Finally, we obtain the following decomposition of the regular character of U. 
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Theorem 6.3. Let p be the regular character ofU. Then, 

WW 



p - 1^ 77 — 1 — V ^ DA 



where the sum is over all basic subsets D C $ and all maps (/>: D —* F g x . 
Proof. Let z G U be arbitrary. Then, 

1^ 77 7 — r £dA z ) - 2^ 77 7 — \ in* I 2^ 'W*) 

^J (WW) ^ (W> W> ^ |<W>I /g ^ 

(W^W/I^WI^/^ 

= EE */(«*)■ 

Since u* is the disjoint union u* = \J D , O^ j, we obtain 

]T ]T 0,(0,) = £ tf,(a z ) = <* a „o|u| = <J M |tf|, 

A0 / eo i,4> / eu * 

and the result follows. D 

7. Irreducible characters of maximum degree 

As a final remark, we observe that the description of the irreducible characters of 
maximum degree of U as given in [7J Section 6] remains valid for arbitrary odd primes. 
The proofs given there can be adapted (and simplified) using the results of the present 
paper and also some properties of the Kirillov functions associated with coadjoint U- 
orbits. Given an arbitrary [/-orbit O C u*, we define the Kirillov function <po'- U — > C 
by the rule 



for all z G U (see [T5l Section 5] for the similar definition in the case of finite algebra 
groups). In fact, it can be shown that every irreducible character of maximum degree is 
precisely the Kirillov function associated with a (unique) coadjoint [/-orbit of maximum 
cardinality. We should mention that similar results have been obtained recently by J. 
Sangroniz in the paper [23], where the author uses Kirillov's method of coadjoint orbits 
and shows that, for sufficiently large orbits, the associated Kirillov functions are in fact 
irreducible characters. In this section, we shall use Sangroniz's results to resume the 
description given in our previous paper [TJ. 
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We start by considering the symplectic case U < Sp 2ri (g). Let 

T = {2si : 1 < i < n} U {Ei + e i+1 : 1 < i < n} C $. 

Then, for any basic subset DCT and any map 0: D — > F x , the supercharacter £r>,<i> is 
irreducible (by Corollary I6.2p . In particular, in the case where, either D, or DU {2e n }, is 
a maximal basic subset of T, then ^d,<j> is irreducible and has maximum degree g n ( n_1 )/ 2 
(see the proof of [3, Proposition 6.3]). On the other hand, it is easy to see that the number 
d n of all these pairs (D, (f>) can be computed by the "Fibonacci" recurrence relation 

= q, d 2 = q 2 - 1, 

= (q — l)(dn-i + dn-2), for n > 3. 

Therefore, by [231 Theorem 12], we obtain the following result. 

Theorem 7.1. Let x be an irreducible character ofU< Sp2 n {q)- Then, x has maximum 
degree if and only if x — £d,4> where, either D, or D U {2e„}, is a maximal basic subset 

ofT and <fi: D — ► F x is any map. 

Next, we consider the even orthogonal case U < 02 n (q)- Let 

T = {ei + e i+ i: 1 < i < n}, 

D C T be a basic subset, and <fi: D — ► F x be a map. Then, by Corollary 16.21 we easily 
conclude that (£d,<j>,£d,<j>) = Q'' D ', and a repetition of the proof of [7, Proposition 6.5] 
shows that £d,0 is multiplicity free; hence, it has g' D ' irreducible constituents each with 
degree equal to g~' D '££) >? i(l). In particular, for 

D = \Ei + E2, £3 + £4, ■ ■ ■ , £2r-l + e 2r} 

where r = [^J , the supercharacter £0,0 has q r (distinct) irreducible constituents, each 
with degree equal to q^ n ^ where 

, . J n(n — 2)/2, if n is even, 
\(n-l) 2 /2, if n is odd. 

On the other hand, if n = 2r is even and 

D = {Ei+ £ 2 , £3 + £4, • • • , ^2r-3 + ^2r-2} C T, 

the supercharacter ^^ has q r ~ x (distinct) irreducible constituents, each with degree equal 
to g n ( n ~ 2 )/ 2 . Therefore, for the basic subset 

D = {e 1 + £ 2 , £3 + £4, ■ ■ ■ , £2r-3 + £2r-2J U {^r-1 ~ ^2r} Q $ 

and any map 0: D — > F x , the supercharacter £d j( £ also has g r ~ x (distinct) irreducible 
constituents, each with degree equal to g™(™- 2 )/ 2 . Now, by Theorem 12.11 we niay repeat the 
proof of [3 Proposition 6.6] to conclude that q^ n ^ is the maximum degree of an irreducible 
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character of U, and thus we have obtained d n irreducible characters of maximum degree 
where 

r g („+2)/2( g _ ^(n-2)^ if n ig even5 

n ~ \q^V\q - 1)(«-D/ 2 , if n is odd. 
Using [23l Theorem 13], we conclude the proof of the following result. 

Theorem 7.2. Suppose that U < O^nio), an d let x be an irreducible character of U. 
Let D C $ be the (unique) basic subset and 0: D — > F x the (unique) map such that 
(x,£d,<j>) 7^ 0. Then, the following holds. 

(i) If n is even, then x has maximum degree if and only if 

D = {e l + £ 2 , e 3 + e 4 ,..., £ n _ 3 + e n _ 2 } U D x 

where D x C {e n -i - e n , e n -i + e n }. 
(ii) If n is odd, then x has maximum degree if and only if 

D = {6i + E2, £3 + £4, . . . , S n -\ + £„}. 

Finally, we consider the odd orthogonal case U < 02n+i(?)- Let 

T = {ei + Ei+i: 1 < % < n}, 

let D C T be a basic subset, and let 0: D — > F x be a map. Then, as in the even case, 
we conclude that, for 

D = {e 1 + £ 2 , £3 + £4, • • • , ear-i + ^2r} Q T 
where r = [^J , the supercharacter ^ D ^ has g r (distinct) irreducible constituents, each 
with degree equal to g n ( n_1 )/ 2 . On the other hand, using Corollary 16.21 (see also [3, 
pg. 423]), we conclude that, for the basic subset 

D = I ^ £l + £2 ' £ 3 + £ 4, • • • , £n-3 + £n-2J U {£„-i}, ii u = 2r is even, 
I {£1 + £ 2 , £3 + £ 4 , • • • , £„- 2 + £n-i} U {£„}, if n = 2r + 1 is odd, 

and any map (f>: D —* F g x , the supercharacter ^ D ^ has, either g r ~ 1 , or q r , (distinct) 
irreducible constituents, each with degree equal to g n ( n_1 )/ 2 . Finally, as in the even case, 
we conclude that g n ( n_1 )/ 2 is the maximum degree of an irreducible character of U, and 
thus we have obtained d n irreducible characters of maximum degree where 

J g (n-2)/2^ gH _ 1 ^ g _ 1 ^n/2 ) [f n i s eV en, 
n ~ |g(«+l)/2( g _ 1 )(n-l)/2 ) if n is odd _ 

Using [23l Theorem 13], we conclude the proof of the following result. 

Theorem 7.3. Suppose that U < 02n+i(<?); and let x be an irreducible character of U . 
Let D C $ be the (unique) basic subset and <fi: D — > F g x the (unique) map such that 
{x->£,d,cj>) 7^ 0. Then, the following holds. 
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(i) // n is even, then x has maximum degree if and only if 

D = {e 1 + e 2 , e 3 + e 4 , • • • , e„_ 3 + £ n _ 2 } U D ± 

where, either D\ = {e„_i + e n }, or D\ = {e n }. 
(ii) // n is odd, then x has maximum degree if and only if 

D = { £l + e 2 , e 3 + e 4 , . . . , e n - 2 + e„_i} U £>i 

where D 1 C {& n }. 

References 

[1] Andre, C. A. M. Irreducible characters of the unitriangular group and coadjoint orbits. PhD thesis, 

University of Warwick, 1992. 
[2] Andre, C. A. M. Basic characters of the unitriangular group. J. Algebra 175, 1 (1995), 287-319. 
[3] Andre, C. A. M. Basic sums of coadjoint orbits of the unitriangular group. J. Algebra 176, 3 

(1995), 959-1000. 
[4] Andre, C. A. M. The basic character table of the unitriangular group. J. Algebra 241, 1 (2001), 

437-471. 
[5] Andre, C. A. M. Basic characters of the unitriangular group (for arbitrary primes). Proc. Amer. 

Math. Soc. 130, 7 (2002), 1943-1954. 
[6] Andre, C. A. M. Hecke algebras for the basic characters of the unitriangular group. Proc. Amer. 

Math. Soc. 132, 4 (2004), 987-996. 
[7] Andre, C. A. M., and Neto, A. M. Super-characters of finite unipotent groups of types B n , C n 

and D n . J. Algebra 305, 1 (2006), 394-429. 
[8] Andre, C. A. M., and Neto, A. M. A supercharacter theory for the Sylow p-subgroups of the 

finite symplcctic and orthogonal groups. Preprint (2008). 
[9] Andre, C. A. M., and Nicolas, A. P. Supercharacters of the adjoint group of a finite radical 
ring. J. Group Th. (to appear). 
[10] Andre, C. A. M., and Pinho, O. Factorization of supercharacters of finite algebra groups. 

Preprint (2008). 
[11] Arias-Castro, E., Diaconis, P., and Stanley, R. A super-class walk on upper-triangular 

matrices. J. Algebra 278, 2 (2004), 739-765. 
[12] Carter, R. W. Simple groups of Lie type. Pure and Applied Mathematics. Wiley, London, 1972. 
[13] Carter, R. W. Finite Groups of Lie type (conjugacy classes and complex characters). Pure and 

Applied Mathematics. Wiley, New York, 1985. 
[14] Curtis, C. W., and Reiner, I. Methods of representation theory (with applications to finite groups 

and orders). Vol. II. Pure and Applied Mathematics. Wiley, New York, 1987. 
[15] Diaconis, P., AND Isaacs, I. M. Supercharacters and superclasses for algebra groups. Trans. 

Amer. Math. Soc. 360, 5 (2008), 2359-2392. 
[16] Gudikov, P. M., Kapitonova, Y. V., Polyak, S. S., Rudko, V. P., and Tsitkin, A. I. 

Classes of conjugate elements of a unitriangular group. Cybernetics 26 (1990), 47-57. 
[17] Huppert, B. Character theory of finite groups, vol. 25 of de Gruyter Expositions in Mathematics. 

Walter de Gruyter, Berlin, 1998. 
[18] JACOBSON, N. Lie algebras. Dover, New York, 1979. 
[19] KAZDHAN, D. Proof of Springer's hypothesis. Israel J. Math. 28 (1977), 272-286. 



29 



[20] Kirillov, A. A. Variations on the triangular theme. In Lie groups and Lie algebras: E. B. Dynkin's 

seminar, vol. 169 of Amer. Math. Soc. Transl. Ser. 2. Amer. Math. Soc., Providence RI, 1995, 

pp. 43-73. 
[21] Marberg, E., and Thiem, N. Superinduction for pattern groups. Preprint (2008). 
[22] Sangroniz, J. Characters of algebra groups and unitriangular groups. In Finite groups 2003. Walter 

de Gruyter, Berlin, 2004, pp. 335-349. 
[23] Sangroniz, J. Irreducible characters of large degree of Sylow p-subgroups of classical groups. 

Preprint (2008). 
[24] Thiem, N., AND Venkateswaran, V. Restricting supercharacters of the finite group of unipotent 

uppertriangular matrices. Preprint (2008). 
[25] Yan, N. Representation theory of the finite unipotent linear groups. PhD thesis, University of 

Pennsylvannia, 2001. 

(C. A. M. Andre) Departamento de Matematica, Faculdade de Ciencias da Universidade 
de Lisboa, Campo Grande, Edificio C6, Piso 2, 1749-016 Lisboa, Portugal 

(A. M. Ncto) Instituto Superior de Economia e Gestao, Universidade Tecnica de Lisboa, 
Rua do Quelhas 6, 1200-781 Lisboa, Portugal 

(C. A. M. Andre & A. M. Neto) Centro de Estruturas Lineares e Combinatorias, Complexo 
Interdisciplicar da Universidade de Lisboa, Av. Prof. Gama Pinto 2, 1649-003 Lisboa, 
Portugal 

E-mail address: caandre@fc.ul.pt 

E-mail address: ananeto@iseg.utl.pt 



